| Kä po 


Dick Snippe - Dutch Public Broadcasting Organizatic 


Geography of the Netherlands 


Geography of the Netherlands 


It's small; 41.543 km? =16.000 mi? About Massachusetts 
+ Connecticut; not even 1% of the USA land area 

But quite densely populated; 17 million inhabitants; 
that's like cramming all of Texas into West Virginia 

It's all flat. No mountains. Easy to lay cables. 

Ideal for good internet connectivity. 

98% of households has broadband internet 

4G coverage is about 98% 

Amsterdam is the main internet hub 


Internet topology 


e ams-ixis the main hub. 
e twolarge cable providers 
e place your gear in Amsterdam and you're all set! 
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Loadbalancing 


Classical loadbalacing: 1 (or a HA pair) loadbalancer; all traffic 
(upstream+downstream) passes through the loadbalancer 


This is fine for most websites / api's 


Loadbalancing 


Loadbalancing 


Classical loadbalacing: 1 (or a HA pair) loadbalancer; all traffic 
(upstream+downstream) passes through the loadbalancer 


This is fine for most websites / api's 


Not so good for streaming where upstream traffic is 1000x 
the downstream traffic. The loadbalancer becomes the choke 
point when is has to handle >10Gbit of traffic 


Loadbalancing 
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Loadbalancing 


A nice approach is "Direct Routing"; IPVS-DR. A linux kernel 
module that acts as a loadbalancer. 


Only the downstream traffic flows through the loadbalancer. 


Uptream traffics goes directly from server to client. 


Loadbalancing 


Loadbalancing 


A nice approach is "Direct Routing"; IPVS-DR. A linux kernel 
module that acts as a loadbalancer. 


Only the downstream traffic flows through the loadbalancer. 
Uptream traffics goes directly from server to client. 


Problem: for large volumes the loadbalancer still drowns in 
the downstream traffic rates (200.000 pps (mostly ACKs)) 


Loadbalancing 


Loadbalancing 


Our solution: PMLB "Poor Mans Load Balancing" 


Not a loadbalancer but a redirector. Can be written in 
programming language of choice. 


e Request comes in at the redirector 

e Theredirector has knowledge about a pool of streaming 
servers and redirects the client to one of them 

e Then, the client has a direct 1-on-1 TCP connection to 
the chosen streaming server 


Loadbalancing 


Client 


3. Get content 
4. Reply: the datagiream 


Ask for redirect 
Answer: "See Node 2" 
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Node 1 Node 2 Node 3 Redirector 


Report Status 


Loadbalancing 


This redirector can do more stuff besides loadbalancing: 


generate secured links to the streaming servers 

do GeolP checks on incoming requests 

do anti-deeplink verification 

generate statistics 

know about up/down status of streaming servers 
route traffic to specific clusters of streaming servers 
generate "stadium is full" redirects 


Content Protection 


Problem: in HLS/HDS/MSS streaming we want to protect 
both the manifests and the data chunks. So only checking at 
the redirector does not suffice. We have to check each and 
every request. 


Solution: the redirector generates unique redirects to the 
streaming servers. The URL contains a token in which (a.o.) 
the IP address of the client is encoded. 


Content Protection 


If the redirector generates private tokens, the streaming 
servers have to check these tokens for validity 


This is where nginx+lua comes in. 
Very easy to run custom lua code in nginx 
Very easy to write lua code to do the checking 


Token based on shared secret between redirector en 
streaming server 


Content Protection 


Generate a secured redirect: (redirector, php) 


private function maketoken ($clip, Sip, $secret) 


{ 


Sauth-join('|', array(Sclip, Sip, $secret)); 
return md5(S$auth); 


private function redirect (Sclip, Sscheme, Sserver, Sip, $secret) 


{ 
$token = $this->maketoken ($Sclip, $ip); 


return "Sscheme://Sserver/secure/Stoken/Sclip"; 


Content Protection 


local function validate() 


local uri pattern = "^/secure/([^/]*) ([^?]*).*" 
local matches, _,token,clip = 


string. find (ngx.var.request_uri, uri pattern) 
if (matches) then 


ocal p = IO 


local ourtoken = 


ngx.md5(clip .. p .. ip .. p 
if (ourtoken -= token) then 


config.secret) 


ngx.exit(ngx.HTTP FORBIDDEN) 
end 


else 
ngx.exit(ngx.HTTP FORBIDDEN); 
end 
end 


Content Protection 


We use Lua, because... 


Lua is anice, clean, simple to learn, simple to read 
language. 


Same can be achieved using NGINX javascript module 


Javascript knowledge is more widespread than Lua. 


Additional functionality 


GeolP protection 
Use Maxmind geolocation database + php pecl module 


$countrycode = geoip_country_code_by_name($ip); 


Additional functionality 


Anti deeplink verification 


Client generates time-based tokens with shared secret 


Ssecret = "correct horse battery staple"; 
Suri = "/clips/protected/for-your-eyes-only.mp4"; 
Sserver = "content.omroep.nl"; 


$t hex = sprintf("$08x", time()); 
$token = md5 (Ssecret.Suri.St hex); 
Șurl = sprintf ("http://%s/secure/%s/%s%s", 


Sserver, $token, $t_hex, Suri); 


Additional functionality 


route traffic to specific clusters of streaming servers 


e Each network has its own AS number 

e Maxmind database exists to lookup AS numbers by IP. 

e Redirector recognizes specific AS numbers and redirects 
clients to a cluster specific to that AS number 


Additional functionality 
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SSL Optimizations 


The challenge: 


Most websites use https nowadays 

Streaming used to be a separate protocol (e.g. rtsp), but 
nowadays (HLS,HDS, MSS) it's http(s) 

Your browser sees that as part of the website 

Soit complains about mixed content if streaming is done 
over http instead of https 

Net result: streaming servers have to do https 


SSL Optimizations 


SSL is very CPU intensive 


Streaming is usually network bound. With SSL switched on 
that shifts to CPU bound. 


NGINX is already great for running doing SSL, but there are 
some tweaks to increase efficiency 


SSL Optimizations 


Netflix has the best solution: 


e push SSL handling to the kernel 

e now network traffic is again zero copy. l.e. the outgoing 
data needs to be touched only once and not multiple 
times for encryption 

e works on FreeBSD 


But... we use Linux. 


SSL Optimizations 


Linux optimizations 


run a new kernel! 

lots of info on the internet 

smp_affinity 

receive packet steering (on old hardware) 
codel 


SSL Optimizations 


Openssl optimizations 
e add support for chacha-poly ciphers 
NGINX optimizations 


e pcre-jit 


/server-status in Lua 


Commercial variant has a nice status (ngx_http_api_module) 


Open Source nginx server-status is very terse: 


Active connections: 523 

server accepts handled requests 
2217414 2217414 4736545 

Reading: 0 Writing: 234 Waiting: 289 


Can we do better using a bit of lua code? 


/server-status in Lua 


Sure! 


Add a hook for when the request starts and when it's done: 
rewrite by lua block { s = require "stats"; s.incoming() y 


log by lua block { s = require "stats"; s.outgoing() } 


/server-status in Lua 


Use dicts to keep persistent state: 
local log dict- = ngx.shared.log diet 


function m.outgoing() 


iner(log dict, "bytes sent", ngx.var.bytes sent) 
incr CLOG dot, "toust rege", 1) 


end 


Add any other logic you wish 


/server-status in Lua 


Add a hook to display the counters: 


location /server-stats 


{ 


content by lua block { s = require "stats"; s.showstatus() } 
include /e/lib/ip/omroepnet.nginx; 
include /e/lib/ip/beheer.nginx; 


deny all; 


/server-status in Lua 


Display the default counters 


local function stub status (total reqs) 

-- Active connections: 6112 

-- server accepts handled requests 
224956831 224956831 614701793 


end 


-- Reading: 0 Writing: 14 Waiting: 6096 
local total reqs - log dict:get("total reqs") or 0 
ngx.say("Active connections: ", ngx.var.connections active) 
ngx.say("server accepts handled requests") 
ngx.say(0, " ", 0, " ", total reqs) 
ngx.say("Reading: ", ngx.var.connections reading, 

" Writing: ", ngx.var.connections writing, 

" Waiting: ", ngx.var.connections waiting) 


/server-status in Lua 


Display the custom counters 


local function extended status () 


ngx.say("") 

ngx.say("time ", os.date("$Y$m$d %H:%M:%S", now)) 

ngx.say("server pid ", ngx.var.pid) 

ngx.say("lua version ", VERSION) 

ngx.say("nginx version ", ngx.var.nginx version) 

ngx.say("bytes sent ", log dict:get("bytes sent") or 0, " ", normalize("$ 
log dict:get("bytes sent") or 0)) 

ngx.say("http reqs ", log dict:get("http reqs") or 0) 

ngx.say("https reqs ", log dict:get("https reqs") or 0) 


end 


/server-status in Lua 


Example results 


Active connections: 528 

server accepts handled requests 

0 0 4735057 

Reading: 0 Writing: 252 Waiting: 275 


time 20180924 10:37:09 

server pid 5103 

lua version Lua 5.1 

nginx version 1.15.3 

bytes sent 15247571421763 13.87T 
http reqs 475262 

https reqs 4259795 


Sliding averages: 


counter five sec /s 
bytes_sent 271.21M 54.24M/s 
total reqs 67 13.40/s 
http regs 5 1.00/s 


https reqs 11 2.20/s 


one min 
877.74M 
232 
69 
333 


/s 
14.63M/s 
3.87/s 
1.15/s 
5.55/s 


one_day 
1.30T 
430169 
35566 
394574 


/s 
15.78M/s 
4.98/s 
0.41/s 
4.57/s 


